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Abstract 

The purpose of this paper is to review and critique a groundbreaking research report. The 
research article under review in this paper is: "A Big Apple for Educators: New York City's 
Experiment with Schoolwide Performance Bonuses: Final Evaluation Report". In 2007-08, a 
program called the Schoolwide Performance Bonus Program (SPBP) was held in the New York 
City public schools on behalf of the New York City Department of Education (NYCDOE) and 
United Federation of Teachers (UFT). They established a memorandum of understanding (MoU) 
with the RAND Corporation and Vanderbilt University evaluated the program to assess the 
SPBP. The objective was to see the students’ development outcome based on changed 
compensation plans for teachers hence motivating the teachers to perform well in order to 
receive rewards. The idea is for the collaboration and bonus to boost morale, therefore resulting 
in better outcomes. Result of the study showed that there was no evidence to suggest that student 
achievement increased as a result of SPBP. 
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Critique of "A Big Apple for Educators: New York City's Experiment with Schoolwide 
Performance Bonuses: Final Evaluation Report" 

The article "A Big Apple for Educators: New York City's Experiment with Schoolwide 
Performance Bonuses: Final Evaluation Report" is a comprehensive empirical research, 
evaluating New York City’s experiment with a pay-for-performance program, known as 
Schoolwide Performance Bonus Program, hereafter referred to as SPBP. The research was 
conducted by RAND Education (part of Rand Corporation) and the National Center on 
Performance Initiatives at Vanderbilt University. The purpose of the study was to examine the 
implementation of SPBP, and whether this program improved student academic achievement in 
New York City public schools. Schools were randomly divided into intervention or comparison 
groups. The intervention group of schools were given the option to participate or opt out of the 
program. Participating schools joined the program voluntarily. SPBP was then implemented in 
about 200 high-needs schools (grades K-12) in the years 2007 to 2010.Schools that chose not to 
participate in SPBP did so for philosophical as well as practical reasons, such as working 
relationships in the school. The program was sponsored by the New York City Department of 
Education (NYCDOE) and United Federation of Teachers (UFT), who commissioned the 
evaluation report from the researchers. SPBP’s “Theory of Action“ (which can be seen in the 
appendix) hypothesized financial and group incentives would improve motivation, staff 
collaboration, cooperation, recruitment, and retention. This would then translate into an 
improvement in the school and in classroom performance. Consequently, these improvements 
would lead to higher student achievement (Marsh, Springer, McCaffrey, Yuan, Epstein, Koppich, 
Kalra, DiMartino, & Peng, 2011). This sounds intuitively plausible. Theoretically, staff and 


teachers’ anticipatory motivation to improve their schools, and the morale boost post receipt of a 
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bonus could lead to better performing schools, and higher student achievement. Each 
participating school was eligible to receive $3000 per full-time union represented staff. In order 
to receive these funds, schools had to meet their annual performance target. SPBP allowed 
schools to distribute bonuses as they saw fit, and many chose egalitarian methods of 
disseminating these funds. After a two-year study, the evaluators concluded that SPBP did not 
improve student achievement in any grade level, or improve School Progress Reports. They also 
concluded that the program did not have an effect on “teacher reported attitudes, perceptions, and 
behaviors” (Marsh, et al., 2011, xxvi). 

The methodology used to conduct the evaluation and assessment of SPBP was a mixed 
qualitative and quantitative method. Design of this study was an experimental design. The 
authors identified the following questions as their main research questions: “1. How was the 
SPBP program implemented? 2. What were the intermediate outcomes of the program? 3. How 
did the program affect student performance?” (Marsh, et al., 2011, p. 81).They had sub questions 
under each main question. Researchers analyzed administrative data and student test scores from 
participating schools, as well as from the comparison schools. They conducted fieldwork through 
site visits, as well as interviewed teachers, school administrators, staff members, union 
representatives, and stakeholder such as funders. They conducted case studies, and used surveys 
with teachers and school administers. Statistical analysis was used to compare the intervention 
group with the comparison groups, as well as the eligible schools who did not participate in the 
program. 

The methodology used for this study was appropriate, rigorous, and comprehensive. 
Researchers were able to answer all their research questions. Using mixed methods, over a two- 


year period, researchers were able to gain an in depth understanding of the various factors 
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involved in how SPBP was implemented. They were able to identify participants’ perception, 
experiences with the program, and ultimately answer whether SPBP was successful in its goals. 

The authors offer several reasons why SPBP was not successful. They point out that 
monetary bonuses might actually not have a significant motivational power. Many teachers and 
staff saw the bonuses as a reward or acknowledgement of their hard work, rather than as an 
incentive to change their behaviors. Monetary incentives might not have the power to influence 
change, because of other structural barriers to change, such as limited resources, social capital, 
lack of expertise, and school leadership. Researchers also found that some teachers viewed the 
bonuses as added pressure, rather than a net positive. 

The contribution of this research cannot be overstated. NYCDOE initially suspended the 
SPBP program pending the results of this evaluative report. After the report was completed, the 
findings led to the dismantlement of SPBP entirely. While researchers were able to make 
recommendations that could help strengthen SPBP, they could not show that following these 
recommendations would actually change the outcome. This research also contributes to the 
literature on pay-for-performance. It deepens our understanding of factors that need to be in 
place for such programs to succeed. Researchers identified some conditions that are necessary 
for pay-for-performance programs to succeed, such as “understanding of the program”, 
“expectancy”, “valence”, “buy-in”, and“perceived fairness” (Marsh, et al., 2011, xxi). These 
necessary conditions were not all met in every participating school. Not meeting these conditions 
caused misunderstandings about the program’s target and created questions about the fairness 
and transparency. These issues all weakened motivation in participants. This has implications for 
future programs. By first identifying necessary conditions, and then setting up the support 


needed to meet these conditions, future programs can avoid the pitfalls faced by SPBP. 
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Some programs fail because of issues in their design, or in the process of implementation. 
The findings of this research suggests something more significant, namely that the theories 
underlying pay-for-performance programs may be flawed. This is the most significant 
contribution of this study. The results of this research challenge the actual theoretical framework. 
Many businesses use monetary bonuses and incentive programs, such as vacations and gifts to 
motivate higher employee productivity. Public schools are not businesses. The motivations of 
people entering public education are less tangible, and often have more intrinsic value. These 
need to be taken into account when looking for ways to improve school performance. 

Researchers identified some limitations with their study, such as only having one year of 
teacher data, and a 57% response rate. The main limitation of this study is that researchers relied 
exclusively on student test scores to evaluate improvement; other evaluative measures were not 
used. As the authors point out, test scores do not capture a holistic picture of student 
achievement. Future studies on similar programs should take these limitations into account. 

During an educational leader’s discussion of this article, they expressed similar concerns 
as those identified in this research about pay-for-performance programs. While most people 
would appreciate higher pay, monetary incentives tied to school performance feel like added 
pressure, rather than a motivational push to improve behavior. Educators agreed that those 
entering education are motivated by a myriad of personal reasons that are not easily quantifiable. 
They also identified potential pitfalls of such programs, such as “teaching to the test”. 
Furthermore, these leaders argued that there are many barriers to changing behaviors in schools, 


such as school culture. 


CRITIQUEFINAL EVALUATION REPORT 


Reference 
Marsh, J. A., Springer, M. G., McCaffrey, D. F., Yuan, K., Epstein, S., Koppich, J., Kalra, N., 
DiMartino, C., & Peng, A. (2011). A big apple for educators: New York City's experiment 
with schoolwide performance bonuses: Final evaluation report. Santa Monica, 


California: Rand Corporation. 


CRITIQUEFINAL EVALUATION REPORT 


Background on Pay-for-Performance Programs and the New York City SPBP 27 


Figure 2.1 


SPBP Theory of Action 


Bonus 


© Financial 
incentive 


¢ Group 
incentive 


RAND MG1114-7.1 


SPBP Theory of Action 


Improved Improved 
motivation school 
(anticipatory) and 
and morale classroom 
(post-receipt) practices 


Improved 
student 
achievement 


Improved staff 
collaboration 
and cooperation 


Improved staff 
recruitment 
and retention 


